Search appliances, such as are used in conjunction with an intranet, typically receive an initial list of URLs to crawl, crawl those URLs, and create an index based on the documents retrieved. Recrawls are typically performed in sequence—the first page crawled is also the first page recrawled. The result can be that less important pages and pages which change infrequently that appear earlier in the URL list are crawled before other, more important pages. Since a crawler can only crawl a finite number of pages in a given time period, situations can arise where less important or unchanged pages may be crawled to the exclusion of more important or changed pages. There exists a need to be able to schedule when pages should be recrawled.